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(57) ABSTRACT 

A medical operation simulator includes a three-dimensional 
image reconstruction section, an instrument -image forming 
section, and an approach data presenting section. The three- 
dimensional image reconstructing secUon processes data 
representing slice images, thereby generating three- 
dimensional image data. The three-dimensional image data 
is supplied to a display device. The instrument-image form- 
ing section forms an image of a medical instrument. The 
image moves in the same way that a medical instrument 
provided on an operation input device is manipulated by an 
operator. The instrument -image forming section supplies 
data representing the image of the instrument to the display 
device. The display device displays the image of the instru- 
ment together with a three-dimensional image represented 
by the three-dimensional image data generated by the three- 
dimensional image reconstructing means. The approach data 
presenting section converts data showing an approach of the 
medical instrument toward a region of interest into at least 
one of an aural sensation, visual sensation, pressure sensa- 
tion and touch sensation, by performing approach data 
presentation processing, and then presents at least one of the 
aural sensation, visual sensation, pressure sensation and 
touch sensation. 

16 Claims, 5 Drawing Sheets 
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MEDICAL OPERATION SIMULATION 
SYSTEM CAPABLE OF PRESENTING 
APPROACH DATA 

BACKGROUND OF THE INVENTION 

The present invention relates to a medical operation 
simulation system. More particularly, the invention relates to 
a medical operation simulator and a method of controlling 
the simulator. The simulator enables an operator to simulate 
a medical operation by manipulating an imitation instrument 
provided on an operation input device, thereby moving a 
virtual instrument in a virtual^sp ace ,^with r espect to a 
three-dimensional image conste ffl3ornF $fe^Mges 
formed by means of X-ray CT, MRI, MRAor the like, and 
which displays the positional relation between a region of 
interest and each virtual instrument moving as the operator 
operates the operation input device. 

A medical operation may be simulated and navigated by 
using slice images such as X-ray CT images, MRI images or 
MRA images. The simulation and navigation thus performed 
are important since they help operators to plan procedures 
for operations to be performed or to support operators to 
perform operations. 

In the present specification, operative navigation per- 
formed during actual operations will be called "simulation," 
too. 

Simulation and navigation are effected by using slice 
images which have been formed at regular intervals. Impor- 
tant regions and anti-invasive regions, such as the affected 
parts, the blood vessels and the nerve, are extracted from the 
slice images and visualized. The simulation assists the 
operator, enabling him or she to move the virtual instruments 
without touching the virtual anti-invasive regions. 

A method of performing operative navigation is known, 
as disclosed in Jpn. Pat. Appln. No. 8-10266. In the method, 
the positions of the instruments being used in an operation 
are detected, slice images that correspond to the instruments 
are selected, and patterns indicating the positions of the 
instruments are displayed and superimposed on the slice 
images. And an alarm is generated when any instrument 
approaches a critical region. 

To accomplish more realistic simulation it is effective to 
display three-dimensional images. The three-dimensional 
images displayed reduce the visual discrepancy between the 
real regions and instruments, on the one hand, and the virtual 
regions and instruments, on the other. Thus, more correct 
simulation can be carried out. 

In the conventional method described above, medical 
instruments only are displayed and superimposed on the 
slice images. No three-dimensional images are displayed. 
This makes it difficult for operators to grasp the three- 
dimensional relationship between the position of each 
instrument and the position of any region. 

In the conventional method,-an alarm is generated when 
any instrument comes within a predetermined distance from 
an anti-invasive region. However, it is impossible to deter- 
mine minute changes in the distance between the instrument 
and the region. 

BRIEF SUMMARY OF THE INVENTION 

The present invention has been made in view of the 
foregoing. The object of the invention is to provide a 
medical operation simulation system which includes a medi- 
cal operation simulator and a method of controlling the 
simulator. The simulator reconsti 



iniagesgfgffi 

images, enables an operator to operate a operation input 
device to move virtual instruments on a display and presents 
data showing how much any instrument has approached a 

5 region of interest, thereby accomplishing high-reality simu- 
lation. Comprising the simulator and the method of control- 
ling the simulator, the medical operation simulation system 
can present the approach data to the operator. 
To attain the object, a medical operation simulator accord- 

10 ing to an aspcct_pfthe invenUor^comprises;_^^pe£^ 
d^^nsiQ ^iim l^elr^MM^gl^ 
reprlfejLtin * " 



15 



20 



30 



dime^ibn^^Tflage 5 Q*atVt6^1lisplay device; an ^^^pSnVT) 
image forming device for forming an image of a medical 
instrument, the image moving in the same way that a 
medical instrument provided on an operation input device is 
manipulated by an operator, and for supplying data repre- 
senting the image of the medical instrument to the display 
device, thereby to display the image of the medical instru- 
ment together with a three-dimensional imagejrcp resented_- 
by theffiS8l3iile^^ 
d imefisj|g^pin^ 
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sensation, presjmg|Sjnsatiqn and touch sensation by per- 
forming approacn^mf^jiresentation processing, and for 
presenting at least one of the aural sensation, visual 
sensation, pressure sensation and touch sensation. 

To achieve the object, a method of controlling a medical 
operation simulator, according to another aspect of the 
invention, comprises: processing data representing slice 
images, thereby to generate three-dimensional image data, 
and then supplying the three-dimensional image data to a 
display device; forming an image of a medical instrument, 
the image moving in the same way that a medical instrument 
provided on an operation input device is manipulated by an 
operator, and then supplying data representing the image of 
the medical instrument to the display device, thereby to 
display the image of the medical instrument together with a 
three-dimensional image represented by the three- 
dimensional image data generated; and converting data 
showing an approach of the medical instrument toward a 
region of interest into at least one of an aural sensation, 
visual sensation, pressure sensation and touch sensation by 
performing an approach data presentation processing, and 
presenting at least one of the aural sensation, visual 
sensation, pressure sensation and touch sensation. 

The present invention makes it possible to perform an 
accurate simulation, because the operator can know how 
much the image of the instrument has approached, or how 
far it is positioned from, an affected part or a critical region, 
while observing the three-dimensional image displayed. 

Additional objects and advantages of the invention will be 
set forth in the description which follows, and in part will be 
obvious from the description, or may be learned by practice 
of the invention. The objects and advantages of the invention 
may be realized and obtained by means of the instrumen- 
talities and combinations particularly pointed out hereinaf- 
ter. 

BRIEF DESCRIPTION OF THE SEVERAL 
VIEWS OF THE DRAWING 

The accompanying drawings, which are incorporated in 
and constitute a part of the specification, illustrate presently 
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preferred embodiments of the invention, and together with 
the general description given above and the detailed descrip- 
tion of the preferred embodiments given below, serve to 
explain the principles of the invention. 

FIG. 1 is a block diagram showing a medical operation 5 
simulator according to an embodiment of the invention; 

FIG. 2 is a schematic representation of slice images 
formed at regular intervals which have been input via the 
image input device 5 to the three-dimensional dimentional 
image reconstructing device 6 of the arithmetic apparatus 3 10 
and which are to be processed to in order to reconstruct a 
three-dimensional image in the medical operation simulator 
shown in FIG. 1; 

FIG. 3 is a diagram for explaining how a slice image of 
a region of interest in the field of neurosurgery should be 
displayed in order to make it easy to understand which part 
of the image should be the target, while segmentation is 
being performed on the slice image in the medical operation 
simulator shown in FIG. 1; 

FIG. 4 is a flow chart explaining how alarm signals are 2Q 
generated in the case where a plurality of regions of interest 
are set and different threshold distances are designated for 
the regions, each designated in accordance with the impor- 
tance of the region, in the medical operation simulator 
shown in FIG. 1; 25 

FIG. 5 is a graph for how an alarm signal is changed in 
magnitude as an instrument approaches a region of interest, 
in the medical operation simulator shown in FIG. 1, in order 
to inform the operator that the instrument is very near the 
region and still approaching the region; 30 

FIG. 6 shows the screen of the display device 8 of the 
simulator shown in FIG. 1, which displays the distances 
between the instrument, on the one hand, and some regions 
of interest, on the other, so that the operator may visually 
recognize not only the alarm signal but also the distance 35 
between the instrument and each region of interest; 

FIGS. 7A and 7B show slice images formed at appropriate 
intervals and displayed in the direction of incision, in the 
medical operation simulator shown in FIG. 1; 

FIG. 8 is a diagram for explaining how an instrument- 40 
guiding path is set in the three-dimensional image before 
simulating neurosurgery and after reconstructing a three- 
dimensional image of the brain and setting region of interest 
in the three-dimensional image, and how the simulation 
proceeds along the instrument-guiding path in the medical 45 
operation simulator shown in FIG. 1; 

FIG. 9 is a diagram for explaining how the magnification 
and motion of the images of instruments, displayed on the 
display 80, are controlled by signals supplied from the 
operation input device 1, as in the process of deforming the 50 
images, in the medical operation simulator shown in FIG. 1; 

FIG. 10 is a diagram for explaining how medical instru- 
ments 28 and 28 are manipulated within the sheath 27 of an 
endoscope, when an operation using an endoscope is simu- 
lated by using the medical operation simulator shown in 55 
FIG. 1; and 

FIG. 11 is a flow chart explaining how a feedback signal 
is generated to apply a large reaction to the operation input 
device 1 when it is detected that an instrument contacts a 
region of interest, while an operation using an endoscope is 60 
being simulated by using the medical operation simulator 
shown in FIG. 1. 

DETAILED DESCRIPTION OF THE 

INVENTION 65 

Reference will now be made in detail to the presently 
preferred embodiments of the invention as illustrated in the 
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accompanying drawings, in which like reference characters 
designate like or corresponding parts throughout the several 
drawings. 

FIG. 1 is a block diagram showing a medical operation 
simulator according to an embodiment of the invention. 

As shown in FIG. 1, the simulator comprises an operation 
input device 1, a position/direction detecting device 2, an 
arithmetic apparatus 3, an inspection image data input 
device 5, a display device 8, and an alarm signal generating 
device 10, Imitation instruments are provided on the opera- 
tion input device 1. Each imitation instrument can be 
manipulated freely. When an operator manipulates any imi- 
tation instrument, the position/direction detecting device 2 
detects the position and orientation of the imitation instru- 
ment and generates data representing the position and ori- 
entation detected. The data is supplied to the arithmetic 
apparatus 3. 

The arithmetic apparatus 3 comprises an instrument 
image forming device 4, a three-dimensional image recon- 
structing device 6, an image superimposing device 7, and a 
distance calculating device 9. 

The instrument-image forming device 4 processes the 
data supplied from the position/direction detecting device 2, 
thereby generating data showing an image of the imitation 
instrument. The display device 8 displays the image of the 
instrument, which moves on the screen of the display device 
in the same way as the operator manipulates the imitation 
instrument. 

Data representing slice images of the type shown in FIG. 
2, whic h have be enJforrned at regular intervals, is input to 
thga^meticjapparatus-3^£r<0m the inspection image data 
input device 5. In the arithmetic apparatus 3, the three- 
di mfensi onalimage-reconstructingjd^ 
generattng^data-showing an three-dune nsional image to be 
inspected or examined. The data generated by the device 6 
is supplied to the image-siroerim^ 

The image superimposing device 7 synthesizes the three- 
dimensional image data with the instrument-image data 
supplied from the instrument -image forming device 4, 
thereby generating synthesized image data. The synthesized 
image data is supplied to the display device 8. The display 
device 8 displays a synthesized image represented by the 
input data (i.e., a three-dimensional image to be inspected) 
and an image of the imitation instrument, on the same 
screen. 

In the three-dimensional image reconstructing device 6, 
the data items showing the slice image are filtered to achieve 
image smoothing and is subjected to threshold process, 
extraction of characterizing regions and segmentation. Hie 
device 6 combines the data items thus processed, generating 
data representing a three-dimensional image. The data is 
eventually supplied to the display device 8, which displays 
the three-dimensional image represented by this data. 

The display device 8 can display images by means of 
either sti^^rfM^^goryolume rendering, in accordance 
with the instruction input By the operator. Surface rendering, 
which is one of the methods for reconstructing three- 
dimensional images, is performed in the three-dimensional 
image reconstructing device 6. More specifically, the surface 
rendering is accomplished by generating equal-value sur- 
faces at the boundaries between the regions of any image 
that has been subjected ^s^gg egggspjln the segmentation 
the regions are made tc^Se^mTransmittance, so that the 
rendering may be achieved to enable the operator to see 
inner regions. 

If volume rendering is performed, it will become possible 
to display segmentation regions in different colors. 



07/09/2003, EAST Version: 1.03.0002 



US 6,283,763 Bl 

5 6 

In the process of reconstructing a three-dimensional FIG. 4 is a flow chart explaining how the device 10 

image, some regions of interest, such as affected parts and generates an alarm signal. As shown in FIG. 4, the position 

anti-invasion regions (e.g., the blood vessels and the nerve), coordinates of an imitation instrument are input at regular 

are set from the slice images. These regions are subjected to intervals in Step 14. In the loop consisting of Steps 15, 17, 

%gmemation. They are set, either automatically or manu- 5 ig an d 16, the distances between the imitation instrument 

all^Ptethttose of the manual setting, the operator can and all reg ions of interest are calculated in Step 17. In Step 

perform threshold process on the slice images and set 18, the distances calculated are compared with the threshold 

whichever region desired. distances designated for the regions, respectively. If any 

gmher,^%is possible to combine a plurality of images, distance calculated is shorter than the corresponding thresh- 



perils CT images, MRI images, MRA images and SPEC 1Q oW dislance , an alarra signal ^ generated for the region of 

( images, in the same three-dimensional space. The resultant {merest in Step 19 

xScombined image may be displayed on the screen of the ' ' t niein-ioin ^ - 

dfiplay~dcvice^8. sequence of Steps 14, 15, 17, 18, 19 and 16 is 

t^T j- T o i. *i_ j* , • repeated at regular intervals in the medical operation simu- 

The display device 8 may be a three-dimensional image . ; . • 

display which can display not only two-dimensional images 1 c lator according to the present invention, 

(e.g., CRT images and LCD images), but also stereoscopic 1S ^ alarm S1 6 nal m W be gradually increased in magni- 

images. In this case, the distance calculating device 9 keeps tadc as shown m FIG 5 - thercb y 10 inform & c operator that 

calculating the distance between i B&smmBm S&mp ^ imitatioD instrument is approaching a region of interest. 

dmW*te«Sl8!Ki5^^ To 1)6 raore specie- the distance-magnitude curves for 

l^SSSriSTthe dista1S%lu1^^ dis- „ n rc & ioDS 2 aad 3 are set as different functions, as illustrated 

play device 8 and the alarm signal generating device 10. The 20 » FIG. 5. Alternatively, different types of approach data 

display device 8 displays the distance. The alarm signal sl S nals ma y bc set > eacb for one rc S loa of mlerest * 

generating device 10 generates an alarm signal on the basis A function can be set for each region of interest on the 

of the distance. More precisely, the device 10 generates an basis of both tne importance of the region and the size 

alarm signal when the distance between the image of the thereof. For example, as shown in FIG. 5, a function 

imitation instrument and the region of interest becomes 25 represented by a steep curve is set for the region 1, which is 

shorter than a predetermined value. sma11 bul important. As shown in FIG. 5, loo, a function 

Since any region of interest has a volume, its center may represented by a gently slope is set for the region 2, which 

either be the center of gravity or the point in the region that 15 lar 8 e bul not 50 important. 

the operator has designated. The distance the calculating 3Q An alarm signal may be generated, which has a great 

device 9 calculates may therefore be defined as one between magnitude and whose magnitude greatly changes as the 

the tip of the image of the imitation instrument and the operator moves the imitation instrument, 

gravity center of the region, or as one between the tip of the This alarm signal indicates that the region of interest to 

image of the imitation instrument and the designated point which the instrument is approaching is a highly critical one. 

in the region. Alternatively, the distance may be defined as 35 In this case, the operator needs to take much care while 

the shortest one between the surface of the region and the manipulating the imitation instrument, 

image of the imitation instrument. The distances between the instrument, on the one hand, 

The alarm signal is supplied to the display device 8. In and the regions of interest, on the other, are always displayed 

accordance with the alarm signal, the display device 8 on the screen of the display device 8, as is illustrated in FIG, 

changes the color of the region of interest displayed on the ^ 6 The operator can therefore visually recognize not only the 

screen. More precisely, the device 8 changes the hue or alarm signal but also the distance between the instrument 

density of the color, or flashes the entire screen. and each region of interest. 

Alternatively, the alarm signal may be supplied to a speech In the present invention, the display device 8 can display 

generating device (not shown), which generate an oral not only a three-dimensional image by means of surface 

alarm. 45 rendering or volume rendering. But also can it display slice 

Furthermore, the alarm signal generated by the alarm images formed at appropriate intervals, arranged in the 

signal generating device 10 may be fed back to the operation direction of incision, as is illustrated in FIG. 7A. Slice 

input device 1, and a mechanism incorporated in the device images can be so displayed, thanks to the operation of the 

1 may apply a force, a pressure or a touch to the operator. three-dimensional image reconstructing device 6. In the case 

In the simulation of neurosurgery, in particular, segmen- so shown in FIG. 7B, the device 8 displays a series of slice 

tation may be performed on some regions of interest in a images as sub-images and a three-dimensional image as 

slice image as shown in FIG. 3. If so, the critical regions 11 main image, on the same screen. The main image and the 

and 12 that should not be invaded are displayed in one sub-images, displayed on the same screen, help the operator 

fashion, while the affected region 13 is displayed in another to decide which part of the region should be incised and in 

fashion. For example, the affected region 13, or target, may 55 which direction it should be incised, 

be displayed in a specific color, while the critical regions 11 Particularly in simulation of neurosurgery, an instrument- 

and 12 are displayed a different color. The target can guiding path is set after a three-dimensional image is recon- 

therefore be well distinguished from the other regions. This structed and regions of interest are set. Then, the operation 

enables the operator to recognize easily the position of the is simulated along the instrument-guid ing^ath ^ ^ . ^ 

target, i.e., critical target 13. 60 FIG. 8 shows a|^^p^^jggn^||m^3^S^^g 

In the case where a plurality of regions of interest are set, Assume that an affectel^prTO 

different threshold distances are designated for these and two^tical^regions 21 and 22 in the three-dimensional 

regions, respectively, in accordance with the importance of image. (^Ms aise, an instrument-guiding path 23 is set in 

each region. The alarm signal generating device 10 gener- the mre^toerSional image before simulating neurosurgery 

ates an alarm signal when an imitation instrument 65 on the regid^^^^|\ 

approaches any region of interest within the threshold dis- There are variousffcypes of operations, which are to be 

tance designated for that region. simulated In one operation, a doctor performs incision, 
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exposing an affected part present in the patient's body. In 
an other o peration, .the doctor cuts no tissues,„and-inse^^n^> 
ehSSsgjO^i^ 

a^d manirni!ateTthe"ins^ the affected 

parrmrougtTthe endoscopeTThe region in which the instru- 
ment is moved differs in accordance with the type of the 
operation. 

In the first-mentioned operation, the instrument is moved 
in a large region. In the operation using the endoscope, the 
instrument is moved in a small region. 

In a simulated operation, the size of a region in which an 
instrument can be moved is determined in accordance with 
the type of the operation. In the medical operation simulator 
of this invention, too, the size of the region in which the 
operation input device 1 can be moved is selected in 
accordance with the type of the operation simulated. More 
specifically, the region is set to enable the device 1 to move 
for 15 cm at most from its neutral or balanced position if the 
simulated operation is one wherein incision is conducted, 
exposing an affected part. If the simulated opera tion is one 
wherein an endoscope is used, the region is set to enable the 
device 1 to move for 30 cm at most from its neutral or 
balanced position. In either case, the operation input device 
1 cannot be moved, with any part of it not moving outside 
the region thus set. 

The size of the image displayed in a simulated operation 
depends on the type of the operation. Namely, the image 
displayed in the simulation of the operation wherein incision 
is conducted, exposing the affected part, an image of the 
affected part is displayed which has the same size as the 
affected part. And the image displayed in the simulation of 
the operation using an endoscope is one dthexenlarged or 
distorted due to the ab^rfarion-inherenUnjhe.endoscope. A 
distorted image is formed by processing the image data. 
(^Fhus, the simulator can present an image which isS^eryT* 
^similar to one that is really observed though the endoscojx-^ 



to 



The size and motion of the image of the instrument, 
displayed by the display device 8, is changed in accordance 
with the signals supplied from the operation input device 1. 
This is achieved by processing the image data in the same 
way as the data representing the image of the affected part 
is processed as described above. 

In an operation using an endoscope, the view field is 
circular as is represented by a region 24 shown in FIG. 9. 
This is becauseahe view fieldis defined by the cross section 
of the onneEisheaih -fbLlhe-endbscopS^The tips of imitation 
instruments 25 and 26 are displayed in this circular view 
field as is illustrated in FIG. 9. The other part of either 
instrument, which is outside the view field, is represented by 
dotted lines. 

In an operation using an endoscope, a guide sheath is used 
to guide a medical instrument into the patient's body. When 
the instrument happens to contact the endoscope or the guide 
sheath, it may not be further moved or may receive a 
reaction. This situation should be and is simulated in the 
medical operation simulator of this invention. In the 
simulation, the distance between mejjmage.oft be imit ation 
msfrumenrand the image of the endc«cope~6r-guide-sheatb- 
is always monitored. When the instrument contacts the guide 
sheath, the operation input device 1 is disabled from moving 
further outwards, or a large reaction is applied to the 
operation input device 1. 

Assume that two instruments 28 and 29 are manipulated 
in a guide sheath 27 as illustrated in FIG. 10. The distance 
between the inner surface of the sheath 27 and the instru- 
ment 28 and the distance between the inner surface of the 
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instrument 29 are monitored continuously. When it is 
detected that the instrument 28, for example, contacts the 
inner surface of the sheath 27, a feedback signal representing 
a large reaction is supplied to the operation input device 1, 
preventing the device 1 from moving further outwards. 

Assume that, as shown in FIG. 10, an instrument 30 is 
inserted into the patient's body in a different direction from 
the direction in which the sheath 27 is inserted. When it is 
detected that the instrument 30 contacts the sheath 27, a 
reaction is applied to the operation input device 1 in the 
same way as described above. 

How the operation input device 1 is prevented from 
moving further will be explained, with reference to the flow 
chart of FIG. 11. In Step 31, the distance d between the 
image of the instrument and the guide sheath or endoscope 
is calculated. In Step 32, it is determined whether the 
distance d is zero (0). If YES in Step 32, the operation goes 
to Step 33. In Step 33, a feedback signal is generated and 
supplied to the operation input device 1. The other steps 
shown in FIG. 11 are identical to those of generating an 
alarm signal, shown in FIG. 4, and are not described here. 

The present invention as set forth in the foregoing 
description provides the following medical operation simu- 
lators and the following methods of controlling the 
simulators, which operate and achieve specific advantages 
as will be described below. 

(1) A medical operation simulator according to a first 
aspect of the invention, comprising: 

a three-dimensional image reconstructing device for pro- 
cessing data representing slice images, thereby to gen- 
erate three-dimensional image data, and supplying the 
three-dimensional image data to a display device; 
an instrument-image forming device for forming an image 
of a medical instrument, said image moving in the same 
way that a medical instrument provided on an operation 
input device is manipulated by an operator, and for 
supplying data representing the image of the medical 
instrument to the display device, thereby to display the 
image of the medical instrument together with a three- 
dimensional image represented by the three- 
dimensional image data generated by the three- 
dimensional image reconstructing device; and 
an approach data presenting device for converting data 
showing an approach of the medical instrument toward 
a region of interest into at least one of an aural 
sensation, visual sensation, pressure sensation and 
touch sensation by performing approach data presen- 
tation processing, and for presenting at least one of the 
aural sensation, visual sensation, pressure sensation 
and touch sensation. 
The simulator according to the first aspect of the invention 
can perform an accurate simulation, because the operator 
can know how much the image of the instrument has 
approached, or how far it is positioned from, an affected part 
or a critical region, while observing the three-dimensional 
image displayed. 

(2) A medical operation simulator of the type described in 
above paragraph (1), according to a second aspect of the 
invention, in which the approach data presenting device 
causes the operation input device to exert a reaction or stops 
the operation input device to inhibit the medical instrument 
from entering an anti-invasion region, when the medical 
instrument approaches the region of interest or the anti- 
invasion region in a virtual space defined by the three- 
dimensional image. 

The simulator according to the second aspect of the 
invention can perform a realistic simulation. This is because 
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the operation input device exerts a reaction, or a sensation of contacted the inner surface of the sheath of the endoscope 

force, to the operator when the medical instrument during the simulated operation using the endoscope, 

approaches the region of interest or the and -invasion region (7) A method of controlling a medical operation simulator, 

(3) A medical operation simulator of the type described in according to a seventh aspect of the invention, comprising: 
above paragraph (1), according to a third aspect of the 5 processing data representing slice images, thereby to 
invention, in which the approach data presenting device generate three-dimensional image data, and then sup- 
outputs an alarm signal when the medical instrument plyi?g the three-dimensional image data to a display 
approaches a region of interest, said alarm signal having a device; 

magnitude that is changed in accordance with a distance forming an image of a medical instrument, said image 

between the medical instrument and the region of interest or 10 moving in the same way that a medical instrument 

a distance from the region of interest, defined by the provided on an operation input device is manipulated 

operator. by an operator, and then supplying data representing 

The simulator according to the third aspect of the inven- the image of the medical instrument to the display 

tion can perform a more accurate simulation. This is because device, thereby to display the image of the medical 

a sensation changing stepwise in intensity as the instrument 15 instrument together with a three-dimensional image 

approaches the affected part or a critical region is given to represented by the three-dimensional image data gen- 

the operator's hand. Thus, the operator can recognize how erated; and 

much the instrument has approached or moved away from converting data showing an approach of the medical 

the affected part or critical region, or which region the instrument toward a region of interest into at least one 

instrument is approaching or from which part it is moving 20 of an aural sensation, visual sensation, pressure sensa- 

away. tion and touch sensation by performing approach data 

(4) A medical operation simulator of the type described in presentation processing, and presenting at least one of 
above paragraph (1), according to a fourth aspect of the the aural sensation, visual sensation, pressure sensation 
invention, in which the three-dimensional image recon- and touch sensation. 

structing device distinguishes an affected region from anti- 25 The method according to the seventh aspect of the inven- 

invasion regions such as blood vessels, nerves and the like, tion can make the simulator perform an accurate simulation, 

and sets an instrument-guiding path for guiding the medical because the operator can know how much the image of the 

instrument from the skin of the head to an affected part in the instrument has approached, or how far it is positioned from, 

brain, and wherein the approach data presenting device an affected part or a critical region, while observing the 

continuously presents the approach data while the Simula- 30 three-dimensional image displayed, 

tion proceeds along the instrument-guiding path. (8) A method of the type described in above paragraph (7), 

The simulator according to the fourth aspect of the according to an eighth aspect of the invention, in which the 

invention can perform a* more realistic simulation, because operation input device exerts a reaction or is stopped to 

the simulation proceeds along a preset instrument-guiding inhibit the medical instrument from entering an anti- 

path which reaches the affected part. 35 invasion region, when the medical instrument approaches 

(5) A medical operation simulator of the type described in the region of interest or the anti-invasion region in a virtual 
above paragraph (1), according to a fifth aspect of the space defined by the three-dimensional image, 
invention, which further comprises a selection device for The method according to the eighth aspect of the inven- 
selecting an operation using an endoscope or an operation tion can make the simulator perform a realistic simulation, 
including an in^o^^ggedure, and in which the approach 40 This is because the operation input device exerts a reaction, 
data presenting device altlls a region within which the or a sensation of force, to the operator when the medical 
medical instrument is able to move and a mode of displaying instrument approaches the region of interest or the anti- 
the image of the medical instrument on the display device in invasion region 

accordance with which operation has been selected by the (9) A method of the type described in above paragraph (7), 

selection device. 45 according to a ninth aspect of the invention, in which the 

The simulator according to the fifth aspect of the inven- approach of the medical instrument toward a region of 

tion can simulate various types of operations, since it is interest is presented by outputting an alarm signal when the 

possible to change the size of the region within which the medical instrument approaches a region of interest, said 

medical instrument can be moved, in accordance with the alarm signal having a magnitude that is changed in accor- 

type of the operation to simulate. That is, the instrument can so dance with a distance between the medical instrument and 

be moved within a small region to simulate an operation the region of interest or a distance from the region of 

using an endoscope, and within a large region to simulate an interest, defined by the operator. 

operation including an incision procedure. The method according to the ninth aspect of the invention 

(6) A medical operation simulator of the type described in enables the simulator to perform a more accurate simulation, 
above paragraph (1), according to a sixth aspect of the 55 This is because a sensation changing stepwise in intensity as 
invention, in which the approach data presenting device the instrument approaches the affected part or a critical 
determines whether the image of the medical instrument region is given to the operator's hand. Thus, the operator can 
contacts a sheath of an endoscope while an operating using recognize how much the instrument has approached or 
the endoscope is being simulated, and prevents the operation moved away from the affected part or critical region, or 
input device from further moving or applies a large reaction 60 which region the instrument is approaching or from which 
to the operation input device when it is determined that the region it is moving away. 

image of the medical instrument contacts the sheath of the (10) A method of the type described in above paragraph 

endoscope. (7), according to a tenth aspect of the invention, in which the 

The simulator according to the sixth aspect of the inven- step of supplying the three-dimensional image data to a 

tion can simulate an operation using an endoscope realisti- 65 display device includes a step of distinguishing an affected 

caily. This is because the operation input device stops and region from anti-invasion regions such as blood vessels, 

would not move once the image of the instrument has nerves and the like, and then setting an instrument-guiding 
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paib for guiding the medical instrument from the skin of the 
head to an affected pan in the brain, and wherein the step of 
presenting the approach data includes a step of continuously 
presenting the approach data while the simulation proceeds 
along the instrument-guiding path. 5 

The method according to the tenth aspect of the invention 
enables the simulator to perform a more realistic simulation, 
because the simulation proceeds along a preset instrument- 
guiding path which reaches the affected part. 

(11) A method of the type described in above paragraph 
(7), according to an eleventh aspect of the invention, which 
further comprises a step of selecting an operation using an 
endoscope or an operation including an incision procedure, 
and in which the step of presenting the approach data is able 
to alter a region within which the medical instrument is able 

to move and a mode of displaying the image of the medical 15 
instrument on the display device in accordance with which 
operation has been selected. 

The method according to the eleventh aspect of the 
invention enables the simulator to simulate various types of 
operations. This is because it is possible to change the size 20 
of the region within which the medical instrument, in 
accordance with the type of the operation to simulate. That 
is, the instrument can be moved within a small region to 
simulate an operation using an endoscope, and within a large 
region to simulate an operation including an incision pro- 25 
cedure. 

(12) A method of the type described in above paragraph 
(7), according to a twelfth aspect of the invention, in which 
in the step of presenting the approach data it is determined 
whether the image of the medical instrument contacts a 30 
sheath of an endoscope while an operating using the endo- 
scope is being simulated, and the operation input device is 
prevented from further moving or applies a large reaction to 
the operation input device when it is determined that the 
image of the medical instrument contacts the sheath of the 35 
endoscope. 

The method according to the twelfth aspect of the inven- 
tion enables the simulator to simulate an operation using an 
endoscope realistically. This is because the instrument can- 
not be moved further once the image of the instrument has 40 
contacted the inner surface of the sheath of the endoscope 
during the simulated operation using the endoscope. 

As has been described in detail, the present invention can 
provide a medical operation simulation system which 
includes a medical operation simulator and a method of 45 
controlling the apparatus. The simulator reconstructs three- 
dimensional images from slice images, displays the three- 
dimensional images, enables an operator to operate a opera- 
tion input device to move virtual instruments on a display 
and presents data showing bow much any instrument has 50 
approached a region of interest, thereby accomplishing 
high-reality simulation. Including the simulator and the 
method of controlling the simulator, the medical operation 
simulation system can present the approach data to the 
operator. 55 

Additional advantages and modifications will readily 
occurs to those skilled in the art. Therefore, the invention in 
its broader aspects is not limited to the specific details and 
representative embodiments shown and described herein. 
Accordingly, various modifications may be made without 60 
departing from the spirit or scope of the general inventive 
concept as defined by the appended claims and their equiva- 
lents. 

What is claimed is: 

1. A medical operation simulator comprising: 65 
a three-dimensional image reconstructing device that: (i) 
processes data representing slice images, thereby to 
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generate three-dimensional image data, and (ii) sup- 
plies the three-dimensional image data to a display 
device; 

an mstrument-imagc forming device that: (i) forms an 
image of a medical instrument, said image moving in 
the same way that a medical instrument provided on an 
operation input device is manipulated by an operator, 
and (ii) supplies data representing the image of the 
medical instrument to the display device, thereby to 
display the image of the medical instrument together 
with a three-dimensional image represented by the 
three-dimensional image data generated by the three- 
dimensional image reconstructing device; and 

an approach data presenting device that: (i) converts data 
showing an approach of the medical instrument toward 
a region of interest into at least one of an aural 
sensation, visual sensation, pressure sensation and 
touch sensation by performing approach data presen- 
tation processing, and (ii) presents at least one of the 
aural sensation, visual sensation, pressure sensation 
and touch sensation; 

wherein the approach data presenting device outputs an 
alarm signal when the medical instrument approaches a 
region of interest, said alarm signal having a magnitude 
that is changed in accordance with one of: (i) a distance 
between the medical instrument and the region of 
interest, and (ii) a distance from the region of interest. 

2. A simulator according to claim 1, wherein the approach 
data presenting device causes the operation input device to 
one of exert a reaction and stop the operation input device 
so as to inhibit the medical instrument from entering an 
anti-invasion region, when the medical instrument 
approaches one of the region of interest and the anti- 
invasion region in a virtual space defined by the three- 
dimensional image. 

3. A simulator according to claim 1, which further com- 
prises a selection device that selects one of an operation 
using an endoscope and an operation including an incision 
procedure, and wherein the approach data presenting device 
alters a region within which the medical instrument is able 
to move and a mode of displaying the image of the medical 
instrument on the display device in accordance with the 
operation selected by the selection device. 

4. A simulator according to claim 1, wherein the approach 
data presenting device (i) determines whether the image of 
the medical instrument contacts a sheath of an endoscope 
while an operation using the endoscope is being simulated, 
and (ii) at least one of prevents the operation input device 
from farther moving and applies a large reaction to the 
operation input device when it is determined that the image 
of the medical instrument contacts the sheath of the endo- 
scope. 

5. A medical operation simulator comprising: 

a three-dimensional image reconstructing device that: (i) 
processes data representing slice images, thereby to 
generate three-dimensional image data, and (ii) sup- 
plies the three-dimensional image data to a display 
device; 

an instrument -image forming device that: (i) forms an 
image of a medical instrument, said image moving in 
the same way that a medical instrument provided on an 
operation input device is manipulated by an operator, 
and (ii) supplies data representing the image of the 
medical instrument to the display device, thereby to 
display the image of the medical instrument together 
with a three-dimensional image represented by the 
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three -dimensional image data generated by the three- 
dimensional image reconstructing device; and 

an approach data presenting device that: (i) converts data 
showing an approach of the medical instrument toward 
a region of interest into at least one of an aural 5 
sensation, visual sensation, pressure sensation and 
touch sensation by performing approach data presen- 
tation processing, and (ii) presents at least one of the 
aural sensation, visual sensation, pressure sensation 
and touch sensation; 10 

wherein the three-dimensional image reconstructing 
device distinguishes an affected region from anti- 
invasion regions, which include blood vessels and 
nerves, and sets an instrument-guiding path for guiding 
the medical instrument from an external area on a head 15 
to an affected part in a brain, and wherein the approach 
data presenting device continuously presents the 
approach data while the simulation proceeds along the 
instrument-guiding path. 

6. A simulator according to claim 5, wherein the approach 
data presenting device causes the operation input device to 
one of exert a reaction and stop the operation input device 
so as to inhibit the medical instrument from entering an 
anti-invasion region, when the medical instrument 
approaches one of the region of interest and the anti- 
invasion region in a virtual space defined by the three- 
dimensional image. 

7. A simulator according to claim 5, which further com- 
prises a selection device that selects one of an operation 
using an endoscope and an operation including an incision 
procedure, and wherein the approach data presenting device 
alters a region within which the medical instrument is able 
to move and a mode of displaying the image of the medical 
instrument on the display device in accordance with the 
operation selected by the selection device. 

8. A simulator according to claim 5, wherein the approach 
data presenting device (i) determines whether the image of 
the medical instrument contacts a sheath of an endoscope 
while an operation using the endoscope is being simulated, 
and (ii) at least one of prevents the operation input device 
from further moving and applies a large reaction to the 
operation input device when it is determined that the image 
of the medical instrument contacts the sheath of the endo- 
scope. 

9. A method of controlling a medical operation simulator 
comprising: 

processing data representing slice images, thereby to 
generate three-dimensional image data, and then sup- 
plying the three-dimensional image data to a display 5Q 
device; 

forming an image of a medical instrument, said image 
moving in the same way that a medical instrument 
provided on an operation input device is manipulated 
by an operator, and then supplying data representing 55 
the image of the medical instrument to the display 
device, thereby to display the image of the medical 
instrument together with a three-dimensional image 
represented by the three-dimensional image data gen- 
erated; and 60 

converting data showing an approach of the medical 
instrument toward a region of interest into at least one 
of an aural sensation, visual sensation, pressure sensa- 
tion and touch sensation by performing approach data 
presentation processing, and presenting at least one of 65 
the aural sensation, visual sensation, pressure sensation 
and touch sensation; 
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wherein the approach of the medical instrument toward a 
region of interest is presented by outputting an alarm 
signal when the medical instrument approaches a 
region of interest, said alarm signal having a magnitude 
that is changed in accordance with one of: (i) a distance 
between the medical instrument and the region of 
interest, and (ii) a distance from the region of interest. 

10. A method according to claim 9, wherein the operation 
input device one of exerts a reaction and is stopped so as to 
inhibit the medical instrument from entering an anti- 
invasion region, when the medical instrument approaches 
one of the region of interest and the anti-invasion region in 
a virtual space defined by the three-dimensional image. 

11. A method according to claim 9, which further com- 
prises a step of selecting one of an operation using an 
endoscope and an operation including an incision procedure, 
and wherein the step of presenting the approach data com- 
prises altering a region within which the medical instrument 
is able to move and a mode of displaying the image of the 
medical instrument on the display device in accordance with 
which operation has been selected. 

12. A method according to claim 9, wherein in the step of 
presenting the approach data it is determined whether the 
image of the medical instrument contacts a sheath of an 
endoscope while an operation using the endoscope is being 
simulated, and wherein the operation input device at least 
one of is prevented from further moving and applies a large 
reaction to the operation input device when it is determined 
that the image of the medical instrument contacts the sheath 
of the endoscope. 

13. A method of controlling a medical operation simulator 
comprising: 

processing data representing slice images, thereby to 
generate three-dimensional image data, and then sup- 
plying the three-dimensional image data to a display 
device; 

forming an image of a medical instrument, said image 
moving in the same way that a medical instrument 
provided on an operation input device is manipulated 
by an operator, and then supplying data representing 
the image of the medical instrument to the display 
device, thereby to display the image of the medical 
instrument together with a three-dimensional image 
represented by the three-dimensional image data gen- 
erated; and 

converting data showing an approach of the medical 
instrument toward a region of interest into at least one 
of an aural sensation, visual sensation, pressure sensa- 
tion and touch sensation by performing approach data 
presentation processing, and presenting at least one of 
the aural sensation, visual sensation, pressure sensation 
and touch sensation; 

wherein the step of supplying the three-dimensional 
image data to a display device comprises distinguishing 
an affected region from anti-invasion regions, which 
include blood vessels and nerves, and then setting an 
instrument-guiding path for guiding the medical instru- 
ment from an external area on a head to an affected part 
in a brain, and wherein the step of presenting the 
approach data comprises continuously presenting the 
approach data while the simulation proceeds along the 
instrument -guiding path. 

14. A method according to claim 13, wherein the opera- 
tion input device one of exerts a reaction and is stopped so 
as to inhibit the medical instrument from entering an anti- 
invasion region, when the medical instrument approaches 
one of the region of interest and the anti- invasion region in 
a virtual space defined by the three-dimensional image. 
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15. A method according to claim 13, which further 
comprises a step of selecting one of an operation using an 
endoscope and an operation including an incision procedure, 
and wherein the step of presenting the approach data com- 
prises altering a region within which the medical instrument 5 
is able to move and a mode of displaying the image of the 
medical instrument on the display device in accordance with 
which operation has been selected. 

16. A method according to claim 13, wherein in the step 
of presenting the approach data it is determined whether the 



image of the medical instrument contacts a sheath of an 
endoscope while an operation using the endoscope is being 
simulated, and wherein the operation input device at least 
one of is prevented from further moving and applies a large 
reaction to the operation input device when it is determined 
that the image of the medical instrument contacts the sheath 
of the endoscope. 
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